Enhanced extended depth of focusing on biological samples

ABSTRACT

A system and method for constructing a digital composite image of a three-dimensional biological sample. The system includes an optical system that captures images of cells and tissue presented on a specimen slide. The system systematically acquires a stack of images at different segments across the specimen slide. For each segment, the system dynamically calculates an optimal focal plane. Once an optimal focal plane is determined for each of the stacks of images, the system generates a composite image by copying the sharpest objects from each of the optimal focal planes.

FIELD OF THE INVENTION

The present invention generally relates to the field of medicaldiagnostics. More particularly, the present invention pertains toimproved systems and methods for processing digital microscope images tofacilitate detection of cancerous and pre-cancerous tissue and cells.

BACKGROUND OF THE INVENTION

Pathologists typically utilize high-resolution microscopes to examinetissue samples, for example, to identify signs of cancer orpre-cancerous cells. In order to make an accurate and correct diagnosis,the pathologist must see cellular and tissue features in focus under ahigh-resolution microscope. However, high-resolutions microscopes usedby pathologists have limitations which make it difficult to analyzethick biological specimens that have objects of interest on differentplanes.

Specifically, a microscope's lens can only be focused at single point,and there is a finite distance in front of and behind this focal pointthat may be considered sharp. This finite distance is known as the depthof field. As is well known, high-resolution microscopes, such as thoseused by pathologists, have a limited or narrow depth of field. As aresult, objects that appear outside of a given depth of field or focalplane of the microscope are blurred and out of focus, forcing thepathologist to manually and continually alter the focus when viewing athick sample. This limits the productivity of the pathologist and alsoincreases the likelihood he or she will miss a subtle feature that mayappear only in a narrow focal plane.

This limitation is particularly acute in the analysis of thick tissuespecimens (e.g., those that are thicker than the depth of field of amicroscope objective) or uneven tissue specimens, since the entirespecimen cannot be imaged in a single focal plane. The three-dimensionalcharacter of such specimens requires constant refocusing to observecells at various contours of the sample. As a result, the pathologistdoes not see the whole sample in focus, limiting the pathologist'sability to recognize subtle diagnostic features that expand over severalfocal planes.

For example, when obtaining a non-lacerational brush biopsy of a tissue,a brush is used that is sufficiently stiff so as to penetrate tissue. Inthe process of obtaining a full thickness tissue specimen, tissuefragments in addition to single cells and cell clusters are obtained andtransferred onto a microscope slide. The collection of such thickspecimens is described, for example, in U.S. Pat. No. 6,258,044.

Such specimens contain single cells, cell clusters and tissue fragments,and are essentially a hybrid between a cytological smear andhistological sections. Such specimens may be, for example, 20 to 60microns thick. However, the depth of field of a typical 20× microscopewith a 0.75 NA (Numerical Aperture) may be just 4 microns. Thus, suchspecimens cannot be readily imaged and, as a result, conventionalmicroscopy does not present all of the information that a pathologistneeds when making a diagnosis (e.g., an image that is entirely infocus).

The ability to view tissue fragments, in addition to single cells, wouldconfer an advantage to a pathologist in making a diagnosis. For example,intact tissue provides the pathologist with important information abouta tissue's architecture that is not available in cytological smears.This benefit is especially critical in the evaluation ofgastrointestinal tissue, which is a complex tissue containing variouscell types including, for example, glandular, squamous and columnarepithelium.

One solution to the above problem is provided in U.S. Pat. No. 8,199,997(“the '997 patent”). That patent discloses systems and methods thatcompose a two-dimensional image out of a thick, three-dimensional,specimen. This allows a pathologist to capture the information availablefrom a three-dimensional specimen without the drawbacks associated witha conventional microscope. The systems and methods disclosed in the '997patent utilize extended depth of focus (“EDF”) processing techniques. Asdescribed therein, with EDF processing, an automated microscope capturesa set of image slices taken at regular intervals along the z-axis (atthe same location) and then recovers from each slice those pixels thatare in focus to build a single composite image from the in focus pixels.

Although the invention of the '997 patent represents a significantimprovement over conventional microscopy techniques, furtherimprovements in EDF-based imaging systems are needed for imaging thick,semi-transparent biological specimens. In this regard, conventional EDFsystems and methods bring all image elements into sharpness, regardlessof their location in a set of images. These systems and methods may dothis by iteratively traversing a collection of images and identifyingthe sharpest portions of each image. A composite image is then formedusing only the pixels located in the sharpest portions of each image inthe image set. This conventional process works well with non-transparentor opaque objects, where it is not possible to see any objects below thesurface. However, in semi-transparent images objects, such as tissuesamples, objects below the surface are visible to the microscope, whichintroduces additional complexity.

Specifically, conventional EDF systems and methods would bring objectson the surface and those underneath the surface into focus, making itappear that both the upper and lower objects are on the same focal planeand causing objects to appear closer to each other than they really are.Such undesirable image artifacts can cause cells to look crowded andtherefore unhealthy, which could significantly change the diagnosis ofthe area rendered by the pathologist and/or computer system, e.g., frombenign to dysplastic (i.e., pre-cancerous). In this regard, healthytissue will appear to have regular spacing between the cells, whereas incancerous tissue the spacing between cells is highly irregular, or cellsare not uniformly aligned with one another. Conventional EDF also has anatural tendency of decimating the z-relationship between objects toenhance focus, whereas it would be desirable to preserve the spacingbetween nuclei, and thereby the true diagnosis.

Accordingly, systems and methods are needed that create a compositeimage from a collection of images while preserving the spatialrelationship between objects on different planes. Further, systems andmethods are needed that identify the optimal focal plane for acollection of images.

Another problem with conventional EDF systems is that the magnificationchanges as the microscope's objective is moved up and down between focalplanes. In particular, when the objective is moved, new objects willappear in focus, while other objects will becomes less focused. At thesame time, those less focused objects also become smaller due tomagnification changes. As a result, the edges in the image move as thefocus changes and the image correspondingly shrinks, potentially causingthe algorithm to recognize each moving edge as a separate edge at eachfocal plane. This may result in the system splitting a single edge intoa “stair-case” of multiple adjacent edges in the composite EDF image. Insuch circumstances, the “moving” false edges may overwhelm the imageitself and introduce staircase artifacts on the composite EDF images.These artifacts may, for example, appear as white flakes on thecomposite image.

In EDF systems, it is often advantageous to obtain a stack of imagesalong a z-axis with large steps between images (e.g., for increasedspeed in the imaging process). This step size may be close to the depthof field of the system. Therefore, systems and methods are needed thatpreserve object edges, yet allow large steps between images.

Furthermore, systems and methods are needed that take advantage of thevaluable diagnostic information uniquely contained in a brush biopsysample. By way of example, a large number of patients in the UnitedStates and across the globe undergo endoscopy procedures, whereby adoctor observes sections of the upper gastrointestinal tract, the bileduct or other areas of the body using an endoscope. In such procedures,a doctor may perform forceps biopsies and/or brush biopsies to retrievetissue samples for laboratory analysis.

During a forceps biopsy procedure, small sections of tissue are excisedfrom focused areas of the esophagus at given intervals. In a laboratory,the excised tissue segments are sliced with a microtome into flat sheetsfor analysis by a pathologist. As such, a pathologist reviewing theseconventional tissue specimens analyzes substantially flat tissuesections where minimal refocusing is necessary.

During a brush biopsy procedure, on the other hand, a brush biopsyinstrument having stiff bristles is used to sweep a wide area of tissueand obtain a full thickness sample of tissue of the wide tissue area.The biopsy brush removes small tissue segments that are transferred to aspecimen slide substantially intact. Since these tissue segments are notsliced (as described above with respect to forceps biopsies), thenatural architecture of the tissue is maintained. Significantly, thispreserves the en face view of the tissue for observation by apathologist and/or analysis by a computer system (unlike conventionalhistologic tissue preparations where the en face view is destroyed dueto tissue slicing).

The en face view of the tissue confers valuable diagnostic information.For example, cells of the gastrointestinal tract are organized in alattice structure that forms a “honeycomb.” This hexagonal tissuearchitecture is typical of glandular cells in the body, such as bileduct, colon, breast, etc., and of transitional regions where squamousand glandular tissues meet such as esophageal or endocervical cells.

In healthy tissue, evenly-spaced nuclei can be observed forming thehoneycomb appearance. However, in early dysplasia, individual nuclei maybecome slightly enlarged and the normal nuclear cytoplasmic ratioincreases. When this occurs, neighboring nuclei grow closer to oneanother and begin crowding together. In addition, instead of packinginto an organized honeycomb, the nuclei become disorganized and therelationships of cells to each other become haphazard in nature. Thus,the presence or absence of a honeycomb structure and the degree ofcrowding and disorganization are important diagnostic features indetecting early stage disease and discerning between dysplastic andbenign conditions.

Although brush biopsies are able to obtain tissue fragments that retainthe en face view of the tissue and retain the honeycomb for clinicalobservation, constituent cells that form the honeycomb are often locatedin different focal planes, and as such, a pathologist is unable toobserve the honeycomb in-focus. Rather, the pathologist is required toview one or more cells at a first focal distance in isolation, then viewone or more cells at a second focal distance in isolation, and so on.Not only is this manual process tedious, it is also unreliable. In thisregard, the pathologist must remember the relationship and distancebetween the cells at the different focal distances and then mentallypiece together all of the information that he or she has observed. Byway of analogy, rather than viewing a picture of a forest, thepathologist is forced to look at individual trees and try to constructin his/her mind an image of the forest.

Known EDF techniques have not resulted in the honeycomb being properlyimaged, as they fail to preserve the spatial relationship between cellsin thick, semi-transparent samples. As a result, the honeycomb is notclearly imaged and its regularity and potential abnormality is harder toevaluate. In conventional EDF systems, all the nuclei will appear on thesame plane, thereby making the honeycomb appear more crowded, giving thepathologist the false impression that the cells are becoming dysplastic.

Accordingly, improved systems and methods are needed that can create anin-focus view of the honeycomb structure for analysis by a computerand/or a pathologist.

SUMMARY OF THE INVENTION

It is an object of the present invention to provide an EDF system thatgenerates an in-focus composite image of a biological sample wherebydiagnostically important image objects are presented in focus andunderlying objects are de-emphasized.

It is another object of the invention to determine a plurality ofoptimal focal planes for different segments of the biological sample andobtain image objects from the plurality of the optimal focal planes togenerate a digital composite image.

It is yet another objection of the invention to determine whether adistance between cells in a tissue are within a predetermined healthydistance, and when such determination is made, to de-emphasize imageobjects underlying a plane occupied by the cells within thepredetermined healthy distance.

It is another object of the invention to generate an en face image of atissue where constituent cells comprising the tissue are located ondifferent planes.

BRIEF DESCRIPTION OF THE DRAWINGS

The features and advantages of the present disclosure will be more fullyunderstood with reference to the following, detailed description whentaken in conjunction with the accompanying figures, wherein:

FIG. 1 is a schematic cross-sectional view of a representative tissuesample that has been imaged using a conventional EDF system.

FIG. 2 is a schematic cross-sectional view of a representative tissuesample that has been imaged using an embodiment of the enhanced EDFsystem of the present invention.

FIG. 3 is a block diagram of the enhanced EDF system according to anembodiment of the present invention.

FIG. 4 is a flow chart of an embodiment of the enhanced EDF processingmethod according to an embodiment of the present invention.

FIG. 5 is a diagram that shows elements of representative imagescaptured using the enhanced EDF system according to an embodiment of thepresent invention.

FIG. 6 is a chart depicting one step of the method performed by theenhanced EDF system according to an embodiment of the present invention.

FIG. 7 is a chart depicting another step of the method performed by theenhanced EDF system according to an embodiment of the present invention.

FIG. 8A is a schematic side perspective view of a columnar epithelialtissue section.

FIG. 8B is a schematic top view of the tissue section of FIG. 8A,showing a honeycomb pattern.

FIG. 9 shows a schematic side view of a columnar epithelial tissuesection where constituent cells of the tissue section occupy differentplanes.

FIG. 10 shows a schematic side view of a columnar epithelial tissuesection where a series of cells occupy an upper plane and an underlyingcell is occupies a lower plane.

FIG. 11 shows a composite image of a tissue specimen obtained withoutthe enhanced EDF system in accordance with embodiments of the invention.

FIG. 12 shows a composite image of tissue specimen obtained using theenhanced EDF system according to an embodiment of the present invention.

FIG. 13 shows schematic comparative representations of a specimen areawith a “staircasing artifact” and of the same specimen area where theartifact is eliminated according to an embodiment of the presentinvention.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the present invention will now be described withreference to the above-identified figures of the Drawings. However, theDrawings and the description herein of the invention are not intended tolimit the scope of the invention. It will be understood that variousmodifications of the present description of the invention are possiblewithout departing from the spirit of the invention. Also, featuresdescribed herein may be omitted, additional features may be included,and/or features described herein may be combined in a manner differentfrom the specific combinations recited herein, all without departingfrom the spirit of the invention.

As discussed above, conventional EDF systems typically blindly extractthe sharpest pixels from each focal plane when generating a compositeimage. Thus, when such algorithms are applied to thick, semi-transparentbiological specimens, they do not necessarily take into account whichspecific pixels belong to which specific objects, and thus, aresometimes unable to preserve the spatial arrangement of such objects.For instance, where multiple objects or cells are situated in differentplanes (but overlay one another), a composite image generated byconventional EDF systems may appear to represent a single cell, when infact there were several cells stacked on top of each other. This isbecause the spatial relationship between the objects in different planesis not always preserved when conventional EDF is used. This issue cansignificantly change the diagnosis of the area rendered by thepathologist and/or computer system, e.g., from benign to dysplastic(i.e., pre-cancerous).

FIG. 1 demonstrates the operation of conventional EDF systems. Specimen1 is a semi-transparent tissue sample having a depth D and containsobjects of interest (e.g., cells) 10, 20 and 30. Objects 30 are locatedon a first focal plane (closest to the top of the specimen), objects 20are located on a second, lower focal plane, and objects 10 are locatedon a third focal plane that is below the second focal plane. Objects 20located in the second focal plane overlap with objects 10 located on thethird focal plane. The output of a standard EDF system is shown incomposite image 5. As can be seen, conventional EDF systems blindlyextract the pixels corresponding to the sharpest objects in each focalplane when generating the composite image. Conventional EDF plane 6 doesnot take into account or preserve the spatial relationship of thevarious objects in the specimen 1. As a result, objects 10 and 20 appearcrowded together in composite image 5, even though they are, in fact,located on different planes. These objects may incorrectly appear as asingle cell or mass in the composite image 5, which could result in anincorrect diagnosis by a pathologist or a computer system. In thisregard, the pathologist may interpret the composite image 5 asdysplastic, when it merely includes healthy cells located on differentplanes. Therefore, it would be desirable if an EDF system could providea composite EDF image with objects 20 and 30 in focus, but objects 10out of focus.

The operation of the enhanced EDF system of the present invention isdemonstrated with reference to FIG. 2 , which shows the same schematicspecimen as in FIG. 1 . Specifically, specimen 1 is a semi-transparenttissue sample having a depth D and contains objects of interest (e.g.,cells) 10, 20 and 30. Objects 30 are located on a first focal plane(closest to the top of the specimen), objects 20 are located on a secondfocal plane (beneath objects 30), and objects 10 are located on a thirdfocal plane (beneath objects 20). Objects 20 overlap objects 10.However, rather than blindly copy the pixels corresponding to thesharpest objects in each focal plane to the composite image 5, theenhanced EDF system identifies optimal focal plane 7 in specimen 1. As aresult, when composite image 5 is generated, objects 10 are deemphasizedand the spatial representation of objects 10 with respect to objects 20is preserved.

Sample Collection and Preparation:

Although applicable to many fields, it has been found that the systemsand methods of the present invention are useful in the analysis oftissue samples collected using a brush biopsy instrument, for othersmear preparations, and for traditional histological samples imaged at40X with a high-NA objective. As discussed above, when obtaining a brushbiopsy of a tissue, a brush is used that is sufficiently stiff so as topenetrate the various layers of tissue (e.g., epithelial tissue). In theprocess of obtaining a full thickness tissue specimen, tissue fragmentsin addition to single cells and cell clusters are collected.

Typically, in the preparation of a cellular specimen for pathology, aclinician will transfer and affix cells and/or tissue to a glassmicroscope slide. The slide is then sent to a laboratory for furtherprocessing and medical diagnosis. Further processing may includestaining the slide to enhance the contrast of the sample (or specificfeatures of a sample) when viewed under a microscope. Such stains mayinclude, for example, Feulgen, Papanicolaou, hematoxylin and eosin(H&E), alcian blue, and THC stains. A laboratory technician may alsoapply a cover slip and a label to the slide. Among other things, thelabel may identify the type of stain applied to the sample. Thisinformation may be represented in a bar code or embedded in anelectronic tracking device (e.g., RFID). As discussed further below, inlater processing steps, a computer system can read this information todetermine the optimum processing algorithm to apply to a particularsample.

In the present invention, however, a slide may undergo additionalprocessing prior to being examined by either a pathologist and/or acomputer system. Specifically, captured digital microscope images of thecellular specimen are further processed by the enhanced EDF systemdescribed herein, which produces an enhanced digital image thatpreserves diagnostically important objects and their spatialrelationships to one another. This increases the accuracy of thecomputer analysis system as artifacts and false images are reduced andthe diagnostically important objects of interest are presented to thecomputer in focus.

A block diagram of the enhanced EDF system 100 of the present inventionis shown in FIG. 3 . The system 100 comprises an optical system 40 forobtaining a collection of images from a slide. The optical system 40 mayinclude a high powered microscope, a slide positioning stage and acamera. A computer apparatus 44 controls the movement of the stage inthe z-direction to obtain a sufficient number of images slices tocompose an image of a particular x-y position. The system 100 furthercomprises a storage device 42 for storing the collection of images.Storage device 42 may comprise a hard drive or SSD (solid state drives)or other type of high speed memory device. The computer apparatus 44 (ormultiple computers working together) processes the collection of z-stackimages in order to generate the enhanced composite image discussedherein. The computer apparatus 44 may utilize specialized imageprocessing hardware (such as a graphical processing unit or “GPU”) forincreased processing speed,

It will be understood by those of ordinary skill in the art that theoptical system 40 may be configured to capture and store an image afterevery move of the stage, or it can alternatively be configured tocapture images consecutively and continuously at regular time intervalswhile the stage moves at a constant speed. In embodiments of theinvention, the latter method may be faster at creating z-stacks.However, care must be taken to add sufficient light into the system(e.g., via a stroboscope) so that the image capture integration time canbe kept to a minimum. In other embodiments of the invention, the systemis configured to perform either a lossy or non-lossy compression on thez-stack images and move the compressed version of the z-stack imagesoffline (e.g. over ethernet) for more intensive EDF calculations. Thisis done so the image z-stack capture process can occur at max speed(constrained by mechanical movements), where the EDF processing can beperformed, in parallel, by multiple computers. This decoupling allowsmax throughput with maximum scalability at minimal cost. All mechanicalmovements are isolated to the scanner/image part, whereas the secondpart is highly scalable by adding additional computers as necessary towork on the individual z-stacks in a round-robin fashion.

One embodiment of processing steps performed by an enhanced EDF systemis shown in the flowchart of FIG. 4 .

Image Collection:

As shown in Step S1 of FIG. 4 , an optical system obtains a collectionof images slices, each taken at different focal depths (or focal planes)along a z-axis (i.e., the microscope axis). The collection of images arepreferably stored in memory of the computer where the CPU has directaccess to the computer memory to do the intensive EDF operations, or inanother embodiment the z-stack images can be stored in a separateprocessing or grabber board that has a separate CPU, RISC, GPU or FPGAprocessor that is capable of doing fast EDF operations. Once EDF iscomplete, the collection of images are stored in a data storage device(FIG. 3, 42 ) for retrieval by the computer apparatus. The number ofimages in the stack may depend on a number of factors, including thethickness of the sample under examination, and the depth of field of themicroscope objective. Generally, it is preferred to use an interval lessthan the depth of field of the microscope objective to meet the criteriaof oversampling. This ensures consistent sharpness throughout the fullthickness of the specimen. For example, assuming that a sample is 60 umthick and an image is taken at 4 um focal depth intervals, a total of 15or more images (or slices) may be collected.

The sampling interval can be pre-determined, e.g., based uponpre-established data. Alternatively, the sampling interval can bedetermined dynamically by the computer system, e.g., by measuring thenumber of sharp pixels on each focal plane and adapting the processingwhen relatively few sharp pixels are found. The algorithm may be adaptedby terminating the z-scan prematurely or extending the z-scan ifadditional sharp pixels are still to be found, or by increasing thez-distance between focal planes if minimal sharp pixels are found. Thefewer steps that can be taken the faster the system can present thefinal EDF image, but this has to be balanced with the image quality lossthat can occur if the steps are too large.

Pre-Processing:

In one embodiment, the images in the stack are converted to grayscale,as shown in Step S2 of FIG. 4 . It has been found that by convertingeach image to grayscale, the amount of image data and associatedprocessing is significantly reduced without sacrificing diagnosticaccuracy. In this regard, grayscale conversion may be optimized forspecific immunostains, e.g., for H&E or Alcian blue. For instance, colordeconvolution can be used to enhance immunostained cells and ensure thatparticular colors remain in focus. In one embodiment, the systemautomatically reads information relating to the applied stains from theslides to determine the optimum processing algorithm to apply to aparticular sample.

In another embodiment, rather than convert the collected images tograyscale, the enhanced EDF system performs the EDF processing directlyon the color images. For example, edge contrast can be calculateddirectly from the three RGB color images as the maximum of the red,green and blue contrast.

In Step S3, the various data structures that will be required for imageprocessing may be initialized. These may include a number oftwo-dimensional arrays, including the Max Sharpness Array and Z-IndexArray, which will be discussed further below. Alternative datastructures known to those of skill in the art, such as collections,tables or data objects, may be used in place of pixel arrays.

Locating the Sharpest Objects in the Image Collection:

With reference to Steps S4, S5 and S6 of FIG. 4 , the enhanced EDFsystem iteratively traverses the collection of images taken fromdifferent focal distances along a z-axis in order to identify thelocation of the sharpest objects in the collection of images.Specifically, beginning at the top (or bottom) of the image stack, thesystem initializes by calculating the sharpness of the objects on thefirst image plane. The calculated sharpness values are stored in the MaxSharpness Array and the Z-Index Array is populated with the index of thestarting image plane (e.g., Plane 1, representing the topmost plane).For all subsequent planes along the z axis, the system calculates thesharpness for each pixel or object in that plane and compares thesharpness of the objects in the new image plane with those stored in theMax Sharpness Array. If a new maximum sharpness value is located, thesystem stores: (1) the new maximum sharpness value in the Max SharpnessArray (in place of the old Max value) and (2) its location (i.e., theindex of the image slice in which it is located) in the Z-Index Array.

Determining Optimal Focal Plane:

The enhanced EDF system next calculates the optimal focal plane for thesample under review (Step S7). As discussed above, this step ensuresthat the spatial relationship between objects of interest within thesample is maintained. As a result of obtaining the optimal focal planeand creating a composite image using the derived optimal focal plane,overlying objects are presented in focus and underlying objects aremaintained out of focus.

In an embodiment of the invention, the optimal focal plane is determinedby calculating the distance between cells and determining whether or notthe cells are within a normal or healthy distance from one another(referred to as the “h-distance,” FIG. 2 ). In embodiments of theinvention, the healthy distance, or “h-distance” is a predetermineddistance between cellular edges or between two nuclei. For example, inone embodiment of the invention, the h-distance is calculated bymeasuring a distance between the edges of nuclei of neighboring cells,in another embodiment the h-distance is calculated by measuring thedistance between the centers of neighboring nuclei.

In the event that two cells are determined to be within the h-distance,the system concludes that the two cells are of the same tissue and, assuch, the plane occupied by the neighboring cells will be the focalplane, and underlying cells will remain out of focus. If, however, thedistance between two cells is greater than the h-distance, the systemwill shift the focal plane to allow both, unrelated cells to bemaintained in focus.

For example, referring to FIG. 2 , the system determined that thedistance between objects 20 (e.g. distance A) are within the h-distance.Thus, the plane occupied by objects 20 is selected as the optimal focalplane for that slide segment and objects 20 are presented in focus.Underlying objects 10, on the other hand, remain out of focus.Conversely, distance B between object 20′ and object 30′ is determinedto be greater than the h-distance. As a result, the focal plane shifts(rightward in the orientation shown) to the slide segment where objects30 are positioned within the h-distance from one another. The h-distancemay be measured by linear metric units or by numbers of pixels,according to embodiments of the invention.

In one embodiment of the invention, the focal plane is determined byperforming a “closing” on the Z-Index Array. A closing is set ofoperations where a predefined number of grayscale dilations is followedby an equal number of grayscale erosions. For example, assuming anh-distance of five pixels, the system utilizes a structuring element offive pixels, or it performs multiple iterations to cover the h-distance.Thus, the dilations will completely cover the h-distance gap. Once thegap is filled and pixels on either side of the gap become fused, anysubsequent erosions will not have any effect. If, however, the gap isnot filled, the subsequent erosions will restore the edges to theiroriginal positions. Thus, the closing can fill the gap completely, (i.e.yield the same z-index) and, thus not bring an underlying image (e.g. acell nucleus) to the surface if such image exists between the gap. If,however, the closing does not fill the gap, and there is one or morenuclei underneath between the gap, it will bring the nuclei to the stopof the surface. It will be understood that erosions and dilations may beperformed by any of various techniques known in the art, e.g., theGil-Kimmel dilation/erosion algorithm (See Gil, J. Y., & Kimmel, R,Efficient dilation, erosion, opening, and closing algorithms. IEEETransactions on Pastern Analysis and Machine Intelligence, 24(12),1606-1617 (2002)).

Thus, in the exemplary embodiment shown in FIG. 2 , distance B betweencell 20′ and 30′ is greater than the h-distance. As such, although thedilations will extend the image of cell 20′ in all directions and alsoextend the image of cell 30′ in all directions, the gap between therespective cells will not be filled in. As a result, after the erosionsare performed, the original edges of cells 20′ and 30′ will be restoredand the plane occupied by cells 20 will not fuse with the plane occupiedby cells 30. Instead, the focal plane effectively shifts from the planeoccupied by cells 20 to the plane occupied by cells 30. Conversely,because the distance between cells 20 are within the h-distance, thedilations performed on cells 20 will have the effect of filling the gapsbetween respective cells 20, which will not be reversed by thesubsequent erosions. Thus, the plane occupied by cells 20 will bedetermined as the focal plane and, consequently, cells 20 will bepresented in focus, whereas cells 10 bellow the gaps between cells 20will be presented out of focus. This ensures that the spatialrelationship between cells 20 and lower lying cells 10 are preserved.

In one embodiment, a flat 5×5 approximately circular kernel is used forthe erosions and dilations. In another embodiment, a grayscale gaussiankernel is used, such as that taught in the Gil-Kimmel reference citedabove. The number of erosions and dilations are selected to presentoverlying objects in focus and maintain underlying objects out of focus.Because the determination of an optimal focal plane is made in responseto distances between objects, the optimal focal plane may vary as thesystem moves across the distance of a specimen, concentrating more onthe upper nuclei layer where the nuclei are most visible and has thesharpest features (light is less diffracted near the surface of thesemi-transparent medium) but still capable of bringing deeper nuclei tothe surface.

The result of the dilation/erosion procedure discussed above is furtherillustrated in FIGS. 5 and 6 . In FIG. 5 , a stack of images 15 isshown, with image I₁₀ positioned on the top of the image stack. ImagesI₅ and I₁ include various objects that are in focus in each of theseimage planes. Objects 20 overlap objects 10 in the z-axis. A portion ofa representative Z-Index Array 11 generated from image stack 15 is shownin FIG. 6 . As can be seen, the Z-Index Array contains the location(i.e., the image plane number) of the pixels corresponding to thesharpest objects in the image stack 15 (FIG. 5 ). Specifically, thelocation of objects 20 are denoted by a “5” and the location of objects10 are denoted by a “1”. If the final composite image were to becompiled from Z-Index Array 11 in FIG. 6 (i.e., as in conventional EDF),objects 10 and 20 would appear as a single object (i.e., objects 10would crowd objects 20). As discussed above, this could result is amisdiagnosis. Thus, the spatial relationship in the composite imageneeds to be preserved in order accurately depict the specimen and tomake a correct diagnosis.

FIG. 7 depicts the representative Z-Index Array of FIG. 6 after a numberof dilations and erosions have been performed. Specifically, the objectslocated on Image I₁ have been successfully deemphasized and the spatialrelationship of the objects has been maintained. As can be seen, onlyobjects 20, denoted by a “5,” remain in the Index 12. Thus, when thefinal composite image is compiled, the pixels that would have beenotherwise retrieved from Image I₁ are retrieved from Image I₅.

In one embodiment, the number of dilations or erosions is equal to theh-distance between nuclei in healthy tissue. As stated above, theenhanced EDF system will deemphasize or not bring into focus lowerobjects if the distance between the nuclei of the upper layer is lessthan h-distance. On the other hand if the distance is larger h-distance,it can be assumed that the two nuclei are not of the same tissue andtherefore can bring any lower level objects into focus safely withoutintroducing the crowding effect discussed above. For example theh-distance could be 5 pixels, or 18 microns.

It has been found that this process effectively locates the optimalfocal plane for a collection of images and eliminates the undesirablecrowding effect. The system described herein may be used to find theoptimal focal plane for cellular structures of interest, such as cellnuclei. However, the system can be adapted to focus on other structuresof interest, particularly cytoplasmic mucus pockets in goblet cellsand/or cell boundaries to enhance detection of honeycomb arrangements ofcells.

It has been found that, to find large, bright mucus areas, the systemmay perform dilations and erosion with larger size kernels, such asthose in the range 10×10 to 20×20 (depending on resolution of theimage). This process produces a Z-Index Array for large, bright highcontrast objects, such as mucin regions found in goblet cells. To findcell boundaries, the algorithm performs a morphological operation toenhance thin dark lines (erosion by a ring structuring element followedby dilation by a solid structuring element of the same size). Thisproduces a Z-Index Array for thin dark lines such as fish-scales at theapical surface of the cell. The three Z-Index arrays may be used tocreate three separate EDF images, allowing a user to see differentcellular structures of interest at different focal planes.Alternatively, the three Z-index arrays may be combined by taking theZ-Index with max sharpness, then smoothing by a 5×5 Gaussian kernel.

Generating Composite Image and Post-Processing:

As shown in Step S8, the system next generates the composite image basedupon the Z-index Array (which now contains the location of the optimalpixels to be included in the composite image) and the originalcollection of images. It should also be understood that multiple imagestacks could be obtained for a single slide, separately analyzed (asdiscussed below), and the resultant composite images stitched togetherto form a single composite image. Or, a single stack of images may beobtained and sent to multiple algorithms, each algorithm looking forspecific features and each algorithm generating a unique compositeimage. For example, a user can select an optimal composite image forgoblets, another composite image for dysplastic cells, and anothercomposite image for honeycomb patterns.

Various post-processing operations (Step S9) may optionally be performedon the composite image. In one embodiment, the post-processing includesa sharpness correction, which makes an object's edges appear morepronounced and aids in diagnosis. In one embodiment, the sharpnesscorrection comprises unsharp masking, which is known to sharpen edgeswithout increasing noise. Generally, unsharp masking uses a blurrednegative image (e.g., a Gaussian blur) to create a mask of the originalimage to identify areas of high and low frequency. The mask is thencombined with the original image, creating an image that is sharper thanthe original image. Further post processing steps include Guided filter,XYZ-dilation, haze removal and Z-interpolation, as discussed below.

Analysis of Honeycomb Structure

As described, brush biopsy tissue collection allows for the collectionof tissue fragments that maintain the en face view of the tissue intact.That is, conventional histology samples are sliced and presented astissue slices to a pathologist, and as such, the pathologist neverobserves the en face view of the tissue. The en face honeycombappearance of the tissue yields important clinical information that isuniquely available with brush biopsy collection. Embodiments of theenhanced EDF system allow for the observation and analysis of a tissue'shoneycomb structure as a whole, even while the constituent cells formingthe honeycomb may occupy several focal planes.

FIG. 8A shows a schematic view of a fragment of glandular epithelialtissue 48. As shown, the tissue is formed of columnar cells (e.g. 50)packed together lengthwise. The cells' nuclei 52 are located at a bottomsegment of the cells 50. The cells 50 are located on a basement membrane53. The apical surfaces of the cells form the tissue surface. Whenviewed under a microscope with the focus at the level of the nuclei, thenuclei appear in a hexagonal pattern. When the focus is on the top, thecell membranes form a hexagonal “fish-scales” pattern. When the focus isbetween the top surface and the nuclei, clear mucin regions of gobletcells can be seen most clearly. Most of the pathologist's observation isfocused at the level of nuclei, though the mucin and fish-scales levelviews are also utilized to assist a pathologist in diagnosis.

FIG. 8B shows a top view of the tissue fragment of FIG. 8A with thefocus level on the cells' nuclei. A regular pattern of cell nuclei (i.e.“the honeycomb”) can be observed.

In three-dimensional brush biopsy tissue preparations, however, thecells forming the honeycomb may be located on different focal planes. Inthis regard, it would be impossible to view the honeycomb in focuswithout creating a composite image of it.

For example, referring to FIG. 9 , nuclei cells 54 and 60 are shown at afirst focal plane P1, nuclei cells 56 are shown on a different focalplane P2, and nuclei of cells 58 are shown on still a different focalplane P3. When the microscope objective is set to view nuclei cells 58in focus, then nuclei of cells 54, 60, and 56 will be out of focus. Whennuclei of cells 56 are in focus, then nuclei of cells 58 and cells 54will be out of focus. In this respect, a pathologist utilizing a manualmicroscope will not be able to view the entire honeycomb pattern infocus. Prior EDF systems do not adequately address this problem becausethey will non-discriminately bring all cell nuclei to the top surface.As such, lower lying nuclei associated with cells or tissue that may beunderlying the honeycomb may be included in the composite image, whichmay result in an image artifact.

The system of the invention, on the other hand, dynamically shifts thefocal plane to capture the best focal plane for each segment of thespecimen, and as a result, the honeycomb structure is imaged in focuseven if its constituent cells are located on multiple focal planes.Moreover, cells that are not associated with the honeycomb will remainout of focus.

For example, still referring to FIG. 9 , the EDF system of the inventionwill dynamically select P1 as the optimal focal plane for section E ofthe specimen, select P2 as the optimal focal plane for section F of thespecimen, select P3 as the optimal focal plane for section G of thespecimen and select P1 as the optimal focal plane for section H of thespecimen. In addition, as stated, the EDF system of the inventioncreates a composite image that deemphasizes features located below thecalculated optimal focal plane. Thus, where an optimal focal plane isdetermined based on the proximity of a series of upper cells, cells thatmay be directly beneath the upper cells will remain out of focus,thereby preserving the spatial relationship between the upper cells andthe lower cells. For example, in FIG. 10 a series of cells 62 are shownwith their respective nuclei occupying focal plane P4. A cell 64 isshown underlying cells 62. However, in the embodiment shown, thedistance between cells 62 are within the h-distance, and as such focalplane P4 is determined by the system to be the optimal focal plane. As aresult, underlying cell 64 will remain out of focus. Significantly,features associated with cell 64 will not be brought to the surface.

A computer analyzing a resultant composite image will be more accurateand sensitive because each of the cells in the honeycomb will bepresented in focus and underlying cells will not cause image artifacts.Similarly, rather than analyzing cells and cluster of cells inisolation, the composite image provides a pathologist a gestalt view ofthe honeycomb. This allows the pathologist to analyze cells and cellclusters in the context of other cells and cell clusters.

FIG. 11 is a composite image of a tissue specimen obtained without theenhanced EDF system. As can be seen, large portions of the image areblurred and out of focus. Notably, the honeycomb structure as a wholecannot be observed.

FIG. 12 is a composite image of tissue specimen obtained using theenhanced EDF system of the present invention. In this image, thehoneycomb structure is in focus. As such, the honeycomb can be morereadily observed and analyzed by a pathologist.

Significantly, since the enhanced EDF system images the entirehoneycomb, a computer system can perform morphological analysis toidentify abnormalities in the sample. For example, it has been foundthat cell nuclei can be distinguished from cytoplasm. Once the nucleihave been isolated, the distance between the nuclei (the h-distance) canbe measured. The computer system can then assess whether the honeycombis normal or abnormal, for example by calculating the mean and standarddeviation h-distances to a nucleus' nearest neighbors, and thencalculating the proportion of nuclei with h-distances outside of therange found in regular hexagonal non-dysplastic tissue. Additionally thehexagonal arrangement can be visualized and evaluated with the focalplane at the level of the cell-boundaries instead of at the level ofnuclei, where the image takes on a regular hexagonal “fish-scales”appearance, without distinct nuclei, as shown in FIG. 11 .

Guided Filter

It is advantageous to preserve the edges of tissues and cellularstructures present in the specimen. However, standard averaging or othernon-discriminatory smoothing techniques are incapable of distinguishingedges. Thus, in embodiments of the invention a novel guided filter isutilized to perform accurate edge-preserving smoothing without shiftingthe xy location of the steep contours in the Z-index Array, which areespecially acute with thick specimens. With any objective that uses amagnification lens, the magnification changes as you move away from thefocal plane. On each z-movement, the object being imaged shrinks,causing false edges to move accordingly. Using a guided filter, emphasisis placed where the true edge is, thereby nullifying the effect of thefalse edge. The guided filter may be applied on greyscale or colorimages. The novel usage of the guided filter in this application smoothsthe Z-index Array, which contains the z-location of the sharpest pixels.This dynamically removes z-index noise and is preferred over othersmoothing techniques.

XYZ-Dilation Algorithm

One potential undesired effect of obtaining multiple images from variousfocal distances is the emergence of artifact edges that may arise witheach focal point. That is, with the acquisition of each EDF image slice,out-of-focus pixels adjacent to the true edge may present as a “false”edge. When a succession of such artifact edges are generated, they maytake on the appearance of a staircase (i.e., a staircase effect). Forexample, FIG. 13 (“BEFORE” image) shows a series of artifact edges 66forming a staircase-like presentation on a composite image. This maydistort the sample and obscure useful, diagnostically important, tissueor cellular features.

In embodiments of the invention, this straircasing issue is addressed bya z-index post-processing algorithm that is designed to eliminate thestaircase artifact edges caused by selection of out-of-focus pixels whenperforming the EDF algorithm steps as described. This is achieved bysuppressing multiple adjacent edges and preserving only the strongestedge. In an embodiment of the invention this is achieved by performingdilation with “carry-along” of z-values. To this end, the system isconfigured to determine an edge and run a routine or algorithm thatplaces the edge strength in the top 8-bits of a 16-bit image, and thez-index of the best focus in the bottom 8-bits. During dilation (e.g.,12×12 dilation), the z-values in the bottom 8-bits are carried alongwith the corresponding edge contrast in the top 8-bits, as a side-effectof the dilation algorithm. As a result, the adjacent weaker edges witherroneous z-indices are replaced by the stronger edges with betterz-indices. The best focus z-index image is then updated using the bottom8-bits from the dilated image.

Thus, FIG. 13 (“BEFORE” image) shows a composite image that wasprocessed with an EDF system that did not include the “carry along”feature as described. As shown, a series of false edges 66 are presentat the tissue edge. In the “AFTER” image, on the other hand, the samespecimen is shown having been post-processed using the “carry-along”algorithm as described. As shown, the staircase artifacts have beeneliminated and a sharp true edge 68 is present.

Haze Removal

Microscope images typically include haze caused by non-focused,scattered light. Haze removal can be performed by estimating the imagehaze, and subtracting the haze from the original images to produceclearer images in which diagnostic information is more readily visible.In one embodiment, the system estimates haze by eroding an R,G,B imagewith a flat 5×5 approximately circular kernel, taking the minimum valueof the erosion, performing guided image filtering to smooth the haze andclipping the haze contrast removed to a maximum value of 32 grey levels(on a scale of 256).

Z-Interpolation

In the described EDF system, there may be a large jump in Z-index valuesfrom one focus level to the next, when in reality, the focus levelchanges smoothly. Such discrepancy may result image artifacts. Toaddress this problem, the EDF system may be configured to perform apost-processing Z-interpolation routine to eliminate such artifacts. Inthis embodiment, the system determines the size of the microscope focusstep, which can be as large as the depth of field of the microscope,e.g., up to 4 microns for a 20× objective lens. The Z-interpolationalgorithm stretches the Z-index array to increase the contrast beforeguided filtering of the Z-index array is performed. This producessmoothed z-indices. The algorithm then interpolates the image intensityusing the two best neighboring focus levels, thereby achieving a smoothimage.

As discussed above, in both telecentric and non-telecentric EDF systems,a single edge may appear as multiple edges in the composite EDF image.It has been found that this problem can be addressed by one or more ofthe above processing steps. In alternative embodiments, the ordering ofthe steps may be altered, for example by applying XYZ-dilation beforeguided filtering.

While this invention has been described in conjunction with theembodiments outlined above, it is evident that many alternatives,modifications and variations will be apparent to those skilled in theart. Accordingly, the exemplary embodiments of the invention, as setforth above, are intended to be illustrative, not limiting. Variouschanges may be made without departing from the spirit and scope of theinvention.

What is claimed is:
 1. A method for generating a composite digital imageof a biological sample from a plurality of images of the biologicalsample, each of the plurality of images taken along a single axis, themethod comprising: (a) identifying a first focal plane, wherein thefirst focal plane comprises a first z distance, for a first collectionof image objects at a first x-y location of the biological sample, and(b) identifying a second focal plane, wherein the second focal planecomprises a second z distance, for a second collection of image objectsat a second x-y location of the biological sample, and (c) identifying afirst optimal focal plane and a second optimal focal plane based on (a)and (b) respectively, and (d) combining image objects from a firstoptimal focal plane and image objects from a second optimal focal planeand generates the composite digital image, and using a computerapparatus to calculate a sharpness value for the image objects in thefirst focal plane and calculate a sharpness value for the image objectsin the second focal plane and generate a two-dimensional map of thelocation of the objects having the highest sharpness values, andidentifying an optimal first focal plane and optimal second focal planefor the first and second collection of image objects by performing oneor more dilations and erosions on the two-dimensional map of thelocation of the image objects having highest sharpness values in thefirst and second focal planes wherein the number of erosions anddilations is proportional to the distance between cell objects ofinterest in healthy tissue, so as to thereby identify a first and asecond optimal focal plane, and wherein image objects located on thefirst optimal focal plane are presented in-focus and image objects belowthe first optimal focal plane are deemphasized, and wherein imageobjects located on the second optimal focal plane are presented in-focusand image objects below the second optimal focal plane are deemphasized,so as to generate a composite digital image of a biological sample. 2.The method of claim 1, wherein one or more of the images in theplurality of images are color images and the computer apparatus convertsthe one or more color images in the plurality of images to greyscaleprior to identifying the optimal focal plane for the plurality ofimages.
 3. The method of claim 1, wherein the composite image includes ahoneycomb structure of the biological sample.
 4. The method of claim 3,wherein the honeycomb structure is substantially in focus.
 5. The methodof claim 1, wherein the location of the objects having the highestsharpness value is a z-index and wherein the first optimal focal planeand second optimal focal plane for the first and second collection ofimage objects are identified by performing a number of dilationsfollowed by an equal number of erosions.
 6. The method of claim 1,wherein the cell objects of interest are cell structures.
 7. The methodof claim 1, wherein the cell objects of interest are cell nuclei.
 8. Themethod of claim 1, wherein the cell objects of interest are cytoplasmicmucus pockets in goblet cells.